It seems there is a security risk to set PermitUserEnvironment to yes.
https://man.openbsd.org/sshd_config#PermitUserEnvironment
https://bugzilla.mindrot.org/show_bug.cgi?id=2317

If you want to change /etc/ssh/sshd_config, you can change it after calling remocolab.setupSSHD() or remocolab.setupVNC() and call !service ssh restart or !service ssh reload to restart sshd or reload sshd_config.

That's exactly what I'm doing.
Anyway, without setting proper env vars, I was not able to use the GPU.